Image processing method and apparatus

ABSTRACT

An image conversion system for converting monoscopic images for viewing in three dimensions including: an input means adapted to receive the monoscopic images; a preliminary analysis means to determine if there is any continuity between a first image and a second image of the monoscopic image sequence; a secondary analysis means for receiving monoscopic images which have a continuity, and analyzing the images to determine the speed and direction of motion, and the depth, size and position of objects; a first processing means for processing the monoscopic images based on data received from the preliminary analysis means or the secondary analysis means; a second processing means capable of further processing images received from the first processing means; a transmission means capable of transferring the processed images to a stereoscopic display system.

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of U.S. patent applicationSer. No. 09/516,326, filed Mar. 1, 2000, which is a continuation ofInternational Application Serial No. PCT/AU98/00716 filed Sep. 2, 1998,the teachings of which are incorporated herein by reference, which inturn claims priority from Australian Serial No. PO 8944 filed Sep. 2,1997, the entire contents of which are incorporated herein by reference.

FIELD OF INVENTION

[0002] The present invention relates generally to stereoscopic imagesystems, and in particular to the synthesis of stereoscopic image pairsfrom monoscopic images for stereoscopic display. The present inventionmay also be directed towards a five module method for producingstereoscopic images, that digitizes a monoscopic source, analyses it formotion, generates the stereoscopic image pairs, optimizes thestereoscopic effect, transmits or stores there and then enables them tobe displayed on a stereoscopic display device.

BACKGROUND ART

[0003] The advent of stereoscopic or three dimensional (3D) displaysystems which create a more realistic image for the viewer thanconventional monoscopic or two dimensional (2D) display systems,requires that stereoscopic images be available to be seen on the 3Ddisplay systems. In this regard there exists many monoscopic imagesources, for example existing 2D films or videos, which could bemanipulated to product stereoscopic images for viewing on a stereoscopicdisplay device.

[0004] Preexisting methods to convert such monoscopic images forstereoscopic viewing do not product acceptable results. Other attemptsin film and video have used techniques to duplicate the stereoscopicdepth cue of “Motion Parallax”. These involved producing a delay for theimages presented to the trailing eye when laterals, left or right,motion is present in the images. Other attempts have used ‘LateralShifting’ of the images to the left and right eyes to provide depthperception.

[0005] However, these two techniques are limited and generally only suitspecific applications. For example, the Motion Parallax technique isonly good for scenes with left or right motion and is of limited valuefor the stereoscopic enhancement of still scenes. The Lateral Shiftingtechnique will only give an overall depth effect to a scene and notallow different objects at varying depths to be perceived at the depthswhere they occur. Even the combination of these two techniques will onlygive a limited stereoscope effect for most 2D films or videos.

[0006] Some existing approaches demonstrate limitations of thesetechniques. When an image has vertical motion and some lateral motionand a delay is provided to the image presented to the trailing eye thenthe result is often a large vertical disparity between the left andright views such that the images are uncomfortable to view. Scenes withcontra motion, such as objects moving left and right in the same sceneare also uncomfortable to view. Certain embodiments of these methodsdefine that when objects of varying depths are present in an image thereis a distinct ‘card board cut-out’ appearance of the objects withdistinct depth modules rather than a smooth transition of objects fromforeground to background.

[0007] In all these approaches no successful attempt has been made todevelop a system or method to suit all image sequences or to resolve theproblem of viewer discomfort or to optimize the stereoscopic effect foreach viewer or display device.

OBJECTS OF THE INVENTION

[0008] There is therefore a need for a system with improved methods ofconverting monoscopic images into stereoscopic image pairs and a systemfor providing improved stereoscopic images to a viewer.

[0009] An object of the present invention is to provide such a systemwith improved methods.

SUMMARY OF INVENTION

[0010] In order to address the problems noted above the presentinvention provides in one aspect a method for converting monoscopicimages for viewing in three dimensions including the steps of:

[0011] receiving said monoscopic images;

[0012] analyzing said monoscopic images to determine characteristics ofthe images;

[0013] processing said monoscopic images based on the determined imagecharacteristics;

[0014] outputting the processed images to suitable storage and/orstereoscopic display systems.

[0015] wherein analyzing of said monoscopic images to determine themotion includes the steps of:

[0016] dividing each image into a plurality of blocks, whereincorresponding blocks on an adjacent image are offset horizontally and/orvertically; and

[0017] comparing each said block with said corresponding blocks to findthe minimum mean square error and thereby the motion of the block.

[0018] An image conversion system for converting monoscopic images forviewing in three dimensions including:

[0019] an input means adapted to receive monoscopic images;

[0020] a preliminary analysis means to determine if there is anycontinuity between a first image and a second image of the monoscopicimage sequence;

[0021] a secondary analysis means for receiving monoscopic images whichhave a continuity, and analyzing the images, to determine at least oneof the speed and direction of motion, or the depth, size and position ofobjects, wherein analyzing of said monoscopic images to determine themotion includes the steps of: dividing each image into a plurality ofblocks, wherein corresponding blocks on an adjacent image are offsethorizontally and/or vertically, and comparing each said block with saidcorresponding blocks to find the minimum mean square error and therebythe motion of the block;

[0022] a first processing means for processing the monoscopic imagesbased on data received from the preliminary analysis means and/or thesecondary analysis means.

[0023] Ideally, the input means also includes a means to capture anddigitize the monoscopic images.

[0024] Preferably the image analysis means is capable of determining thespeed and direction of motion, the depth, size and position of objectsand background within an image.

[0025] In a further aspect the present invention provides a method ofoptimizing the stereoscopic image to further improve the stereoscopiceffect and this process is generally applied prior to transmission,storage and display.

[0026] In yet a further aspect the present invention provides a methodof improving stereoscopic image pairs by adding a viewer reference pointto the image.

[0027] In still yet a further aspect the present invention provides amethod of analyzing monoscopic images for conversion to stereoscopicimage pairs including the steps of: scaling each image into a pluralityof regions; comparing each region of a first image with correspondingand adjacent regions of a second image to determine the nature ofmovement between said first image and said second image.

[0028] Preferably a motion vector is defined for each image based on acomparison of the nature of motion detected with predefined motioncategories ranging from no motion to a complete scene change.

[0029] In yet a further aspect the present invention provides a systemfor converting monoscopic images for viewing in three dimensionsincluding:

[0030] a first module adapted to receive a monoscopic image;

[0031] a second module adapted to receive the monoscopic image andanalyze the monoscopic image to create image date, wherein analyzing ofsaid monoscopic image to determine the motion includes the steps of:dividing each image into a plurality of blocks, wherein correspondingblocks on an adjacent image are offset horizontally and/or vertically,and comparing each said block with said corresponding blocks to find theminimum mean square error and thereby the motion of the block;

[0032] a third module adapted to create stereoscopic image pairs fromthe monoscopic image using at least one predetermined technique selectedas a function of the image data;

[0033] a fourth module adapted to transfer the stereoscopic image pairsto a stereoscopic display means;

[0034] a fifth module consisting of a stereoscopic display means.

[0035] Preferably the first module is further adapted to convert anyanalogue images into a digital image. Also, the second module ispreferably adapted to detect any objects in a scene and make adetermination as to the speed and direction of any such motion.Conveniently, the image may be compressed prior to any such analysis.

[0036] Preferably the third module further includes an optimizationstage to further enhance the stereoscopic image pairs prior totransmitting the stereoscopic image pairs to the stereoscopic displaymeans. Conveniently, the fourth module may also include a storage meansfor storing the stereoscopic image pairs for display on the stereoscopicdisplay means at a later time.

[0037] Advantages

[0038] It will be appreciated that the process of the present inventioncan be suspended at any stage and stored for continuation at a latertime or transmitted for continuation at another location if required.

[0039] The present invention provides a conversion technology with anumber of unique advantages including:

[0040] 1) Realtime or Non-Realtime Conversion

[0041] The ability to convert monoscopic images to stereoscopic imagepairs can be performed in realtime or non-realtime. Operatorintervention may be applied to manually manipulate the images. Anexample of this is in the conversion of films or videos where everysequence may be tested and optimized for its stereoscopic effect by anoperator.

[0042] 2) Techniques Include Stereoscopic Enhancement

[0043] The present invention utilizes a plurality of techniques tofurther enhance the basic techniques of motion parallax and lateralshifting (forced parallax) to generate stereoscopic image pairs. Thesetechniques include but are not limited to the use of object analysis,tagging, tracking and morphing, parallax zones, reference points,movement synthesis and parallax modulation techniques.

[0044] 3) Detection and Correction of Reverse 3D

[0045] Reverse 3D is ideally detected as part of the 3D Generationprocess by analyzing the motion characteristics of an image. Correctiontechniques may then employed to minimize Reverse 3D so as to minimizeviewer discomfort.

[0046] 4) Usage in all Applications—Includes Transmission and Storage

[0047] The present invention discloses a technique applicable to a broadrange of applications and describes a complete process for applying thestereoscopic conversion process to monoscopic applications. The presentinvention describes on the one hand techniques for 3D Generation whereboth the image processing equipment and stereoscopic display equipmentare located substantially at the same location. While on the other handtechniques are defined for generation of the stereoscopic image pairs atone location and their transmission, storage and subsequent display at aremote location.

[0048] 5) Can be Used With any Stereoscopic Display Device

[0049] The present invention accommodates any stereoscopic displaydevice and ideally has built in adjustment facilities. The 3D Generationprocess can also take into account the type of display device in orderto optimize the stereoscopic effect.

BRIEF DESCRIPTION OF FIGURES

[0050] The invention will be more fully understood from the followingdescription of a preferred embodiment of the conversion method andintegrated system and as illustrated in the accompanying figures. It is,however, to be appreciated that the present invention is not limited tothe described embodiment.

[0051]FIG. 1 shows the breakdown into modules of a complete systemutilizing the present invention.

[0052]FIG. 2 shows a possible use of multiple processors with a completesystem utilizing the present invention.

[0053]FIG. 3 shows a flow diagram of Module 1 (Video Digitizing) and thefirst part of Module 2 (Image Analysis).

[0054]FIG. 4 shows the second part of a flow diagram of Module 2.

[0055]FIG. 5 shows the third part of a flow diagram of Module 2.

[0056]FIG. 6 shows the fourth part of a flow diagram of Module 2.

[0057]FIG. 7 shows a flow diagram of the first part of Module 3 (3DGeneration).

[0058]FIG. 8 shows the second part of a flow diagram of Module 3 andModule 4 (3D Media-Transmission & Storage) and Module 5 (3D Display).

DETAILED DESCRIPTION

[0059] The present invention aims to provide a viewer with astereoscopic image that uses the full visual perception capabilities ofan individual. Therefore it is necessary to provide the depth cues thebrain requires to interpret such images.

[0060] Introduction

[0061] Humans see by a complex combination of physiological andpsychological processes involving the eyes and the brain. Visualperception involves the use of short and long term memory to be able tointerpret visual information with known and experienced reality asdefined by our senses. For instance, according to the Cartesian laws onspace and perspective the further an object moves away from the viewerthe smaller it gets. In other words, the brain expects that if an objectis large it is close to the viewer and if it is small it is somedistance off. This is a learned process based on knowing the size of theobject in the first place. Other monoscopic or minor depth cues that canbe represented in visual information are for example shadows,defocusing, texture, light, atmosphere.

[0062] These depth cues are used to great advantage in the production of‘Perspective 3D’ video games and computer graphics. However, the problemwith these techniques in achieving a stereoscopic effect is that theperceived depth cannot be quantified: it is an illusion of displaying 2Dobjects in a 2D environment. Such displays do not look real as they donot show a stereoscopic image because the views to both eyes areidentical.

[0063] Depth Cues

[0064] Stereoscopic images are an attempt to recreate real worldvisuals, and require much more visual information than ‘Perspective 3D’images so that depth can be quantified. The stereoscopic or major depthcues provide this additional data so that a person's visual perceptioncan be stimulated in three dimensions. These major depth cues aredescribed as follows:

[0065] 1) Retinal Disparity—refers to the fact that both eyes see aslightly different view. This can easily be demonstrated by holding anobject in front of a person's face and focusing on the background. Oncethe eyes have focused on the background it will appear as though thereare actually two objects in front of the face. Disparity is thehorizontal distance between the corresponding left and right imagepoints of superimposed retinal images. While Parallax is the actualspatial displacement between the viewed images.

[0066] 2) Motion Parallax—Those objects that are closer to the viewerwill appear to move faster even if they are traveling at the same speedas more distant objects. Therefore relative motion is a minor depth cue.But the major stereoscopic depth cue of lateral motion is the creationof motion parallax. With motion in an image moving from right to left,the right eye is the leading eye while the left eye becomes the trailingeye with its image being delayed. This delay is a normal function of ourvisual perception mechanism. For left to right motion the right eyebecomes the trailing eye. The effect of this delay is to create retinaldisparity (two different views to the eyes), which is perceived asbinocular parallax thus providing the stereoscopic cue known as MotionParallax.

[0067] 3) Accommodation—The eye brings an object into sharp focus byeither compressing the eye lens (more convex shape for close object) orexpanding the eye lens (less convex shape for far object) throughneuromotor activity. The amount and type of neuromotor activity is astereoscopic cue for depth in an image.

[0068] 4) Convergence—Is the response of the eye's neuromotor systemthat brings images of an object into alignment with the central visualarea of the eyes such that only one object is seen. For example, when afinger held at arms length is viewed by both eyes and slowly broughttowards the face, the eyes turn inwards (converge) indicating that thefinger is getting closer. That is, the depth to the finger isdecreasing.

[0069] The eyes convergence response is physiologically linked to theaccommodation mechanism in normal vision. In stereoscopic viewing, whenviewers are not accommodated to the ‘Fixation Plane’ (that to which theeyes are converged), they may experience discomfort. The ‘Plane ofFixation’ is normally the screen plane.

[0070] Overview—5 Module Approach

[0071] The present invention describes a system that is capable oftaking any monoscopic input and converting it to an improvedstereoscopic output. For ease of description this complete system can bebroken down into a number of independent modules or processes, namely:

[0072] MODULE I—Monoscopic Image Input (typically video input)

[0073] MODULE 2—Image Analysis

[0074] MODULE 3—3D Generation

[0075] MODULE 4—3D Media (Transmission or Storage)

[0076] MODULE 5—3D Display

[0077]FIG. 1 shows this top down approach to the stereoscopic conversionprocess, where video or some other monoscopic image source is input,images are analyzed, stereoscopic image pairs are generated, transmittedand/or stored and then displayed on a stereoscopic display. Each Moduledescribes an independent process of the complete system from monoscopicimage input to stereoscopic display. However, it will be appreciatedthat the various modules may be operated independently.

[0078] Applications

[0079] Generally, all five modules are used, from monoscopic image inputto display for a particular application. For example, this system may beused in theatres or cinemas. In such an application the 2D video inputcan take the form of analogue or digital to the video sources. Thesesources would then be analyzed to determine speed and direction of anymotion. The processes would then work in either realtime or non realtimein order to create the 3D images. This can be further optimized throughthe use of borders, parallax modification, reverse 3D analysis, shading,and/or texturing. The 3D images may then be stored or transmitted to a3D display, including shutterglasses, polarizing glasses or anautostereoscopic display.

[0080] This system may also be adapted for use with cable or pay-TVsystems. In this application the 2D video input could be video from aVTR, a laser disc, or some other digital source. Again the 3D Generationand/or optimization can proceed in either real time or non real time.The 3D media module would conveniently take the form of transmission viacable or satellite to enable 3D display on TV, video projector, or anauto stereoscopic display.

[0081] The system may also be used with video arcade games, inmultimedia, or with terrestrial or network TV. Depending on theapplication the 2D video input module may obtain source monoscopicimages from a games processor, video from a laser disc, video from VTR,video from a network, or some other digital storage device or digitalsource or telecine process. The 3D Generation can take place in realtime or non real time, and be generated by computer at a centralconversion site, in a user's computer, on a central processor, or someother image processor. The stereoscopic images, can then be stored onvideo or other digital storage device, prior to distribution to cinemasor transmission by a local network. These stereoscopic images may alsobe transmitted to video projectors via a local transmission, oralternatively via VHF/UHF facilities or satellite.

[0082] The 3D display is dependent on the application required, and cantake the form of an auto stereoscopic display device, a video projectorwith polarizing glasses, a local monitor with shutter-glasses, or aset-top box with suitable viewing glasses.

[0083] Single & Multiple Processors

[0084] The complete system can be operated on a single processor withall five modules being processed together or individually in realtime ornon-realtime (Modules 2, 3 and 4). Modules 2 and 3 can be furthersegmented to suit a multitasking or multiprocessor environment, as canbe seen in FIG. 2 for example.

[0085] The use of multiple processors can also be configured to theapplication on hand. For example, modules 1 and 2 could be handled by afirst processor, and modules 3 to 5 by a second processor. If desired,the first processor of this arrangement could be used as a look-aheadprocessor, and the second processor could generate the stereoscopicimages after a delay. Alternatively, a first processor could be used toreceive realtime video, digitize the video and transfer the digitizedvideo to a suitable digital storage device. A second processor, eitheron site or remotely, could then analyze the digitized image and performthe necessary tasks to display a stereoscopic image on a suitabledisplay device.

[0086] Look-ahead processing techniques may be employed to predicttrends in sequences of film or video so that the image processing modesmay be more efficiently selected to optimize the overall stereoscopiceffect.

[0087] The present invention is primarily concerned with the analysis ofmonoscopic images and conversion of the monoscopic images intostereoscopic image pairs together with the optimization of thestereoscopic effect. In this regard the present invention is applicableto a broad range of monoscopic inputs, transmission means and viewingmeans. However, for completeness all five defined modules will bedescribed herein:

[0088] MODULE 1—Image or Video Input

[0089] Module 1 requires that a monoscopic image source or video inputis provided. This source may be provided as either a digital imagesource or an analogue image source which may then be digitized. Theseimage sources may include:

[0090] 1) Analogue Source

[0091] a) Tape based—VCR/VTR or Film.

[0092] b) Disk based—Laser Disk.

[0093] c) Video Camera or other realtime image capture device.

[0094] d) Computer generated images or graphics.

[0095] 2) Digital Source

[0096] a) Tape based—Typical examples are DAT, AMPEX's DCT, SONY'sDigital Betacam, Panasonic's digital video formats or the new DigitalVideo Cassette (DVC) format using 6.5 mm tape.

[0097] b) Disk based storage—Magneto Optical HMO) hard disk (HD),compact disk (CD), Laser Disk, CD-ROM, DAT, Digital Video Cassette (DVC)or Digital Video Disk (DVD) based data storage devices-uses JPEG, MPEGor other digital formats.

[0098] c) Video Camera or other realtime image capture device.

[0099] d) Computer generated images or graphics.

[0100] What is important for the conversion process of the presentinvention is that a monoscopic image source be provided. It is notedthat a stereoscopic image source may be provided which would generallyobviate the need for modules 1 to 3, however, any such stereoscopicimage may be passed through an optimization stage prior to display.

[0101] MODULE 2—Image Analysis

[0102] Referring now to FIGS. 3 to 8 which show flow diagramsdemonstrating a preferred arrangement of the present invention.

[0103] Following reception of 2D images, digitized video or digitalimage data is processed on a field by field or image by image basis inrealtime or non-realtime by hardware, software or by a combination ofboth. Firstly, the image analysis process occurs including the steps of:

[0104] 1) Image compression.

[0105] 2) Motion detection.

[0106] 3) Object detection.

[0107] 4) Motion analysis.

[0108] 1) Image Compression

[0109] Compression of the image is not essential, however, for manyprocesses and applications, compression is a practical optionparticularly, where the processor is not powerful enough to process afull resolution image in the time required.

[0110] Preferably the images are scaled to smaller dimensions. Thescaling factor is dependent on the digital video resolution used foreach image, and is usually defined by the type of image capture facilityused in the digitizing process.

[0111] 2) Motion Detection

[0112] In a preferred embodiment each image may be analyzed in blocks ofpixels. A motion vector is calculated for each block by comparing blocksfrom one image with corresponding blocks from an adjacent image that areoffset horizontally and/or vertically by up to a predetermined number ofpixels, for example ±9, and recording the position that gives theminimum Mean Squared Error.

[0113] For each block, the vector and minimum and maximum Mean SquaredError are recorded for later processing.

[0114] To save on processing time, vectors need not be calculated ifthere is no detail in the block, for example, when the block is ahomogeneous color.

[0115] Other Methods for calculating the motion can be utilized, forexample image subtraction. The present embodiment uses the Mean SquaredError method.

[0116] 3) Object Detection

[0117] An Object is defined as a group of pixels or image elements thatidentify a part of an image that has common features. Thosecharacteristics may relate to regions of similar luminance value(similar brightness), chrominance value (similar color), motion vector(similar speed and direction of motion) or similar picture detail(similar pattern or edge).

[0118] For example a car driving past a house. The car is a region ofpixels or pixel blocks that is moving at a different rate to the,background. If the car stopped in front of the house then the car wouldbe difficult to detect, and other methods may be used.

[0119] A connectivity algorithm may be used to combine the motionvectors into regions of similar motion vectors. An Object may becomprised of one or more of such regions. Other image processingalgorithms, such as edge detection etc, may be used in the detection ofObjects.

[0120] Once Objects are identified in an image they are preferablytagged or given an identification number. These Objects and theirrelevant details (for example position, size, motion vector, type,depth) are then stored in a database so that further processing mayoccur. If an Object is followed over a sequence of images then this isknown as Object Tracking. By tracking Objects and analyzing theircharacteristics they can be identified as being foreground or backgroundObjects and therefore enhanced to emphasize their depth position in animage.

[0121] 4) Motion Analysis

[0122] Once Objects have been detected, the Objects can be analyzed todetermine the overall speed and direction of motion in the image.* Inthe preferred embodiment, this stage determines the type of motion inthe image, and also provides an overall vector.

[0123] By using the Object Detection information and comparing the datato several image motion models a primary determination can be made as tothe best method to convert monoscopic images to stereoscopic imagepairs.

[0124] The image motion models as used in the preferred embodiment ofthe present invention are:

[0125] a) Scene Change.

[0126] b) Simple Pan.

[0127] c) Complex Pan.

[0128] d) Moving Object over stationary background.

[0129] e) Foreground Object over moving background.

[0130] f) No Motion.

[0131] Other motion models may be used as required.

[0132] a) Scene Change

[0133] A scene change as the name suggests is when one image has littleor no commonality to a previous image or scene. It may be detected as avery large absolute difference in luminance between the two images, or alarge difference in the colors of the two images.

[0134] In a preferred arrangement a scene change may be determined whenthe median of the differences of luminance values (0-255) betweenprevious and current images is typically above 30. This value may varywith application but trial and error has determined that this value isappropriate for determining most scene changes.

[0135] A secondary test to determine a scene change can be when thereare too many regions of motion vectors, which appears like random noiseon the image and is likely due to a scene change. This may occur ifthere is a very large amount of motion in the image.

[0136] A third technique to detect a scene change is to analyze the topfew lines of each image to detect a scene change. The top of each imagechanges the least.

[0137] Alternatively, when the majority of motion vector blocks havelarge error values the difference between the two images is too greatand will therefore be considered as a scene change.

[0138] Scene Change and Field Delay

[0139] In the preferred embodiment when there is lateral motion detectedin a, scene the image to the trailing eye is delayed by an amount oftime that is inversely proportional to the speed of the motion. For animage moving right to left the trailing eye is the left eye and for animage moving left to right the trailing eye is the right eye.

[0140] The image sequence delay (or Field Delay) to the trailing eye,may be created by temporally delaying the sequence of video fields tothe trailing eye by storing them in digital form in memory. The currentvideo field is shown to the leading eye and the delayed image to thetrailing eye is selected from the stored video fields depending on thespeed of the lateral motion.

[0141] Over a number of fields displayed, a history as to the change inmotion and change in Field Delays to the trailing eye can be maintained.This helps in smoothing the stereoscopic effect by enabling the imageprocessor to predict any motion trends and to react accordingly bymodifying the delay so that there are no sudden changes.

[0142] If a scene change is detected the Field Delay for the preferredembodiment of the present invention is set to zero to prevent the imagebreaking apart and the Field Delay history is also reset. Field Delayhistory is preferably reset on each scene change.

[0143] b) Simple Pan

[0144] A simple pan describes a lateral motion trend over a series ofimages whereby the majority of analyzed motion is in one direction. Thiswill preferably also cover the situation where the majority of the scenehas a consistent motion, and no stationary objects are detected in theforeground.

[0145] A simple pan can be detected as the major Object having a nonzero motion vector.

[0146] The result of a simple pan is that a positive motion vector isgenerated if the scene is moving to the right (or panning left). In thiscase, the image to the right eye will be delayed. Similarly, a negativemotion vector is generated if the scene is moving to the left (orpanning right). In this case, the image to the left eye will be delayed.

[0147] c) Complex Pan

[0148] A complex pan differs from a simple pan in that there issignificant vertical motion in the image. Therefore, in the preferredembodiment, to minimize vertical disparity between the stereoscopicimage pair sequences, Field Delay is not applied and only ObjectProcessing is used to create a stereoscopic effect. Field Delay historyis stored to maintain continuity with new lateral motion.

[0149] d) Moving Object over Stationary Background

[0150] A moving object over a stationary background is simply thesituation whereby the majority of a scene has no motion; and one or moremoving Objects of medium size are in scene. This situation also resultsin a positive motion vector if the majority of Objects are moving to theright, and a negative motion vector if the majority of Objects aremoving to the left. A positive motion vector produces a delay to theright eye, and a negative motion vector produces a delay to the lefteye.

[0151] In the case where the motion vectors of the Objects in the sceneare not consistent, for example, objects moving to the left and right inthe same scene, then Contra Motion exists and Reverse 3D-correctiontechniques may be. applied.

[0152] e) Foreground Object Over Moving Background

[0153] A Foreground Object over a moving background refers to thesituation where a majority of the scene has motion, and an Object havinga different motion is in the scene, for example a camera following aperson walking. A Background Object is detected as a major Object ofnon-zero motion vector (that is, a panning background) behind an Objectof medium size with zero or opposite motion vector to the main Object,or a major Object of zero vector in front of minor Objects of non zerovector that are spread over the entire field (that is, a largestationary object filling most of the field, but a pan is still visiblebehind it).

[0154] A decision should be made as to whether the foreground Objectshould be given priority in the generation of Motion Parallax, orwhether the background should be given priority. If the backgroundcontains a large variation in depth (for example, trees), then motionvectors are assigned as if a Simple Pan was occurring. If the backgroundcontains little variation in depth (for example, a wall) then a motionvector is assigned that is antiparallel or negative.

[0155] When the background contains a large variation in depth, and amotion vector is assigned to the scene as per Simple Pan methods, thenthe foreground object will be in Reverse 3D, and suitable correctionmethods should be applied.

[0156] f) No Motion

[0157] If no motion is detected such that the motion vectors areentirely zero, or alternatively the largest moving Object is consideredtoo small, then the Field Delay will be set to zero. This situation canoccur where only random or noise motion vectors are determined, or whereno motion information is available, for example during a pan across ablue sky.

[0158] MODULE 3—3D Generation

[0159] Once the images are analyzed they can then be processed to createthe stereoscopic image pairs.

[0160] When viewing a real world scene both eyes see a slightlydifferent image. This is called retinal disparity. This in turn producesstereopsis or depth perception. In other words we see stereoscopicallyby having each eye see a slightly different image of the same scene.

[0161] Parallax on the other hand is defined as the amount of horizontalor lateral shift between the images which is perceived by the viewer asretinal disparity. When a stereoscopic image pair is created, athree-dimensional scene is observed from two horizontally-shiftedviewpoints.

[0162] The present invention utilizes a number of image and objectprocessing techniques to generate stereoscopic image pairs frommonoscopic images. These techniques include:

[0163] 1) Motion Parallax.

[0164] 2) Forced Parallax (Lateral Shifting).

[0165] 3) Parallax Zones.

[0166] 4) Image Rotation about the Y-Axis.

[0167] 5) Object Processing.

[0168] 1) Motion Parallax

[0169] When a scene is moving from right to left, the right eye willobserve the scene first while the left eye will receive a delayed imageand visa versa for a scene moving in the opposite direction. The fasterthe motion the less delay between the images to both eyes. This is knownas motion parallax and is a major depth cue. Therefore, if there islateral motion in a scene, by creating a delay between the images to theeyes a stereoscopic effect will be perceived.

[0170] a) Field Delay Calculation

[0171] Once the nature of the motion in an image has been analyzed andan overall motion vector determined, the required Field Delay can thenbe calculated. Preferably, the calculated Field Delay is averaged withprevious delays to filter out ‘noisy’ values and also to prevent theField Delay changing too quickly.

[0172] As stated above, the faster the motion the less delay between theimage to each eye. Accordingly, smaller values of Field Delay are usedin scenes with large motion vectors, whereas larger delays are used inscenes with little lateral motion. That is, an inverse relationshipexists in the preferred embodiment between the delay and amount ofmotion.

[0173] When a scene change is determined, the history of Field Delaysshould be reset to zero, as if no motion had occurred previously. At thefirst detection of motion when a non zero Field Delay is calculatedwhilst the history of Field Delays is still zero, the entire history ofField Delay is set to the calculated Field Delay. This enables thesystem to immediately display the correct Field Delay when motion isdetected.

[0174] b) Field Delay Implementation

[0175] Motion Parallax can be generated in hardware and software bystoring digitized images in memory. Preferably, the digitized imagescould be stored in a buffer and a single input pointer used with twooutput pointers, one for the left eye image and one for the right eyeimage. The leading eye's image memory pointer is maintained at or nearthe current input image memory pointer while the delayed eyes imagememory pointer is set further down the buffer to produce a delayedoutput. Many images may be stored, up to 8-10 video fields is typical invideo applications. The delay is dependent on the speed of the motionanalyzed in the image. Maximum field delay is when there is minimummotion.

[0176] 2) Forced Parallax (Lateral Shifting)

[0177] Forced parallax can be created by introducing a lateral shiftbetween:

[0178] i) An exact copy of an image and itself

[0179] ii) The two fields of a video frame

[0180] iii) Two frames of a film sequence

[0181] iv) A transformed copy of an image and its original

[0182] A Negative lateral shift is produced by displacing the left imageto the right and the right image to the left by the same amount(establishes a depth of field commencing from the screen plane andproceeding in front of it) and a Positive lateral shift by displacingthe left image to the left and the right image to the right by the sameamount (establishes a depth of field commencing from the screen planeand receding behind it).

[0183] Forced Parallax may be reduced to enhance the stereoscopic effectfor a stationary object in front of a pan, where the object is ‘placed’closer to the screen plane and the background is ‘pushed back’ from thedefined object plane.

[0184] 3) Parallax Zones

[0185] Because most scenes are viewed with the background at the top andthe foreground at the bottom it is possible to accentuate a scene'sdepth by ‘Veeing’ the Forced Parallax. This is done by laterallyshifting the top of the image more than the bottom of an image thusaccentuating the front to back depth observed in a scene.

[0186] Another technique is to use a combination of Motion Parallax andForced Parallax on different parts of the image. For example, bysplitting the image vertically in half and applying different parallaxshifts to each side, a scene such as looking forwards from a movingtrain down a railway track has the correct stereoscopic effect.Otherwise one side would always appear in Reverse 3D.

[0187] 4) Image Rotation about the Y-Axis

[0188] When an object is moving towards the viewer in a real worldscene, the object is rotated slightly in the view for each eye. Therotation effect is more pronounced as the object moves closer.Translating this rotation into the stereoscopic image pairs defines theeffect as follows:

[0189] i) Moving towards the viewer—The left image is rotated verticallyabout its central axis in an anti-clockwise direction and the rightimage in a clockwise direction.

[0190] ii) Moving away from the viewer—The left image is rotatedvertically about its central axis in a clockwise direction and the rightimage in an anticlockwise direction.

[0191] Therefore, by image rotation, the perspective of objects in theimage is changed slightly so that depth is perceived. When thistechnique is combined with Forced Parallax for certain scenes thecombined effect provides very powerful stereoscopic depth cues.

[0192] 5) Object Processing

[0193] Object processing is performed to further enhance thestereoscopic effect, particularly in still images, by separating theObjects and background so that these items can be processedindependently. It is most effective when the objects are large in size,few in number and occupy distinct depth levels throughout the depth offield.

[0194] A database for Object Tagging and Object Tracking can be used toestablish trends so that an Object can be digitally ‘cut out’ from itsbackground and appropriate measures taken to enhance the stereoscopiceffect. Once processing has taken place the Object is ‘Pasted’ back inthe same position on to the background again. This can be termed the‘Cut and Paste’ technique and is useful in the conversion process.

[0195] By integrating the processes of Object Tagging, Tracking, Cuttingand Pasting a powerful tool is available for enabling Object Processingand Background Processing.

[0196] Another Object Processing technique is Object Layering whichdefines an independent depth module for each moving Object. The Objectcan then be placed anywhere on an image because the background filldetail has been defined when the Object was not in that position. Thisis not generally possible with a still Object unless the backgroundfill-in is interpolated.

[0197] A most important issue in stereoscopic conversion is thecorrection of Reverse 3D and Accommodation/Convergence imbalances thatcause viewer discomfort. Object Processing in the preferred embodimentallows corrections to this problem too.

[0198] a) Mesh Distortion and Morphing—This Object processing techniqueallows an Object to be cut and pasted onto a distorted mesh to enhancedepth perception. By distorting an Object in the left eye image to theright and by distorting the same object in the right eye image to theleft, thus creating Object Parallax, the Object can be made to appearmuch closer to a viewer when using a stereoscopic display device.

[0199] b) Object Barreling—This technique is a specific form of MeshDistortion and refers to a technique of cutting an Object from the imageand wrapping onto a vertically positioned half barrel. This makes theObject appear to have depth by making the center portion of the Objectappear closer than the Object edges.

[0200] c) Object Edge Enhancement—By emphasizing the edges of an Objectthere is greater differentiation between the background or other Objectsin an image. The stereoscopic effect is enhanced in many applications bythis technique.

[0201] d) Object Brightness Enhancement—In any image the eye is alwaysdrawn to the largest and brightest objects. By modifying an Object'sluminance the Object can be emphasized more over the background,enhancing the stereoscopic effect.

[0202] e) Object rotation about Y-axis—Object rotation about the Y axisrefers to a similar process to that of image rotation about the Y-axis,except that this time the rotation occurs to the Object only. If theObject in the stereoscopic image pair is ‘Cut’ from its background androtated slightly the change in perspective generated by the rotation isperceived as depth.

[0203] 3D Optimization

[0204] 1) Reference Points or Borders

[0205] When using a normal TV or video monitor to display stereoscopicimages the eye continually observes the edge of the monitor or screenand this is perceived as a point of reference or fixation point for alldepth perception. That is, all objects are perceived at a depth behindor in front of this reference point.

[0206] If the edge of the monitor is not easily seen because of poorambient lighting or due to its dark color then this reference point maybe lost and the eyes may continually search for a fixation point in the3D domain. Under prolonged stereoscopic viewing this can cause eyefatigue and decreased depth perception. A front or rear projectionscreen display system may also suffer from the same problems.

[0207] The present invention therefore preferably also defines a commonborder or reference point within a viewed image. Ideally the referenceplane is set at the screen level and all depth is perceived behind thislevel. This has the advantage of enhancing the stereoscopic effect inmany scenes.

[0208] This reference point can be a simple video border or referencegraphic and, for example, may be of the following types:

[0209] i) A simple colored video border around the perimeter of theimage.

[0210] ii) A complex colored video border consisting of two or moreconcentric borders that may have opaque or transparent sections betweenthem. For example, a 2-3 cm wide mesh border or a wide outer border withtwo thin inner borders.

[0211] iii) A partial border that may occupy any one edge, or any twohorizontal or vertical edges.

[0212] iv) A LOGO or other graphic located at some point within theimage.

[0213] v) A picture within a picture.

[0214] vi) A combination of any of the above.

[0215] What is essential in this embodiment is that the eyes of theviewer be provided with a reference point by which the depth of theobjects in the image can be perceived.

[0216] If a border or graphic is added at the 3D Generation level thenit may be specified to provide a reference point at a particular depthby creating left and right borders that are laterally shifted from eachother. This enables the reference or fixation point to be shifted inspace to a point somewhere behind or in front of the screen level.Borders or graphics defined with no parallax for the left and right eyeswill be perceived at the screen level. This is the preferred mode of thepresent invention.

[0217] A image border or reference graphic may be inserted at the 3DGeneration point or it may be defined externally and genlocked onto thestereoscopic image output for display. Such an image border or referencegraphic may be black, white or colored, plain or patterned, opaque,translucent or transparent to the image background, or it may be staticor dynamic. Whilst a static border is appropriate in most instances, insome circumstances a moving or dynamic border may be used for motionenhancement.

[0218] 2) Parallax Adjustment—Depth Sensitivity Control

[0219] Stereoscopic images viewed through a stereoscopic display deviceautomatically define a depth range (called depth acuity) which can beincreased or decreased by modifying the type and amount of parallaxapplied to the image or objects. It has been found that differentviewers have varying stereoscopic viewing comfort levels based on thedepth range or amount of stereopsis defined by stereoscopic image pairs.That is, while some viewers prefer a pronounced stereoscopic effect witha greater depth range, others prefer an image with minimal depth.

[0220] To adjust the level of depth sensitivity and viewing comfort manytechniques may be used, namely:

[0221] i) Varying the amount of Motion Parallax by varying the FieldDelay

[0222] ii) Varying the amount of Forced Parallax to an image

[0223] iii) Varying the amount of Parallax applied to objects

[0224] By reducing the maximum level of Parallax the depth range can bereduced, improving the viewing comfort for those with perceptionfaculties having greater sensitivity to sterescopy.

[0225] 3) Parallax Smoothing

[0226] Parallax Smoothing is the process of maintaining the total amountof Parallax (Motion Parallax plus Forced Parallax) as a continuousfunction. Changes in Field Delay for specific motion types, that is,Simple Pan and Foreground Object Motion, cause discontinuities in theamount of Motion Parallax produced, which are seen as “jumps” in thestereoscopic images by the viewer. Discontinuities only occur in theimage produced for the trailing eye, as the leading eye is presentedwith an undelayed image. These discontinuities can be compensated for byadjusting the Forced Parallax or Object Parallax in an equal andopposite direction for the trailing eye, thus maintaining a continuoustotal parallax.

[0227] The Forced Parallax or Object Parallax is then adjusted smoothlyback to its normal value, ready for the next change in Field Delay. Theadjustments made to Forced Parallax by Parallax Smoothing are a functionof Field Delay change, motion type and motion vector. To implementParallax Smoothing, the Forced Parallax for the left and right eyeimages should be independently set.

[0228] 4) Parallax Modulation

[0229] The Forced Parallax technique of creating a stereoscopic effectcan also be used to moderate the amount of stereopsis detected by theviewer. This is done by varying the Forced Parallax setting between aminimum and maximum limit over a short time such that the perceiveddepth of an object or image varies over time. Ideally the ForcedParallax is modulated between its minimum and maximum settings every 0.5to 1 second. This enables a viewer to accommodate to their level ofstereoscopic sensitivity.

[0230] 5) Movement Synthesis

[0231] By creating pseudo movement, by randomly moving the background insmall undetectable increments, the perceived depth of foreground objectsis emphasized. Foreground objects are ‘Cut’ from the background, thebackground is altered pseudo-randomly by one of the techniques below andthen the foreground object is ‘Pasted’ back on to the background readyfor display. Any of the following techniques may be used:

[0232] i) Luminance values varied on a pseudo-random basis

[0233] ii) Chrominance values varied on a pseudo-random basis

[0234] iii) Adding pseudo-random noise to the background to createmovement

[0235] 6) Reverse 3D analysis and correction

[0236] Reverse 3D occurs, when the depth order of Objects created byParallax is perceived to be different to that corresponding to the depthorder in the real world. This generally leads to viewer discomfort andshould be corrected. When converting monoscopic images to stereoscopicimage pairs Reverse 3D may be produced by:

[0237] i) Contra motion, objects moving left and right in the sameimage.

[0238] ii) Objects and background moving in different directions.

[0239] iii) Many objects moving at varying speeds Reverse 3D iscorrected by analyzing the nature of the motion of the objects in animage and then manipulating each Object individually using meshdistortion techniques so that the Object Parallax matches with theexpected visual perception norms.

[0240] 7) Miscellaneous Techniques

[0241] By modifying the perspective of an object within an image and byenhancing many of the minor depth cues the stereoscopic effect can beemphasized. The techniques below all operate using the ‘Cut and Paste’technique. That is, a foreground object is ‘Cut’, enhanced and then‘Pasted’ back on to the background.

[0242] a) Shadows—Shading gives an object perspective.

[0243] b) Foreground/Background—By defocusing the background, throughblurring or fogging, a foreground object may be emphasized, whiledefocusing the foreground object the background depth may be emphasized

[0244] c) Edge Enhancement—Edges help to differentiate an object fromits background.

[0245] d) Texture Mapping—Helps to differentiate the object from thebackground.

[0246] Module 4—3D Media (Transmission & Storage

[0247] As for module 1, modules 4 and 5 are not essential to the presentinvention. Module 4 provides for the transmission and/or storage of thestereoscopic images. The transmission means can be adapted for aparticular application. For example the following can be employed:

[0248] 1) Local Transmission—can be via coax cable

[0249] 2) Network TV Transmission—can be via

[0250] i) Cable

[0251] ii) Satellite

[0252] iii) Terrestrial

[0253] 3) Digital Network—INTERNET, etc

[0254] 4) Stereoscopic (3D) Image Storage

[0255] An image storage means may be used for storage of the image datafor later transmission or display and may include:

[0256] i) Analogue Storage—Video Tape, Film, etc

[0257] ii) Digital Storage—Laser Disk, Hard Disk, CD-ROM, MagnetoOptical Disk, DAT, Digital Video Cassette (DVC), DVD.

[0258] Module 5—3D Display

[0259] As for the transmission means the display means can be dependenton the application requirements and can include:

[0260] 1) Set-Top Box

[0261] A set-top box by definition is a small box of electronics thatreceives, decodes, provides accessories interfaces and finally hasoutputs to suit the application. It may incorporate the following:

[0262] a) Video or RF receiver.

[0263] b) Stereoscopic (3D) decoder to provide separate left and rightimage outputs to Head Mounted Devices or other stereoscopic displayswhere separate video channels are required.

[0264] c) Resolution Enhancement—Line Doubling/Pixel interpolation.

[0265] d) Shutter or Sequential Glasses Synchronization.

[0266] e) Stereoscopic depth sensitivity control circuitry.

[0267] f) Accessories interface—remote control with features such as a2D/3D switch and Depth control.

[0268] g) Audio interface—audio output, headphone connection.

[0269] h) Access channel decoding—cable and pay TV applications.

[0270] i) Video or RF outputs.

[0271] 2) Stereoscopic Displays

[0272] Use special glasses or head gear to provide separate images tothe left and right eyes including:

[0273] a) Polarizing glasses—Linear and Circular polarizers.

[0274] b) Anaglyphic glasses—Colored lenses—red/green, etc.

[0275] c) LCD Shutter glasses.

[0276] d) Color Sequential Glasses.

[0277] e) Head Mounted Devices (HMD)—Head gear fitted with two miniaturevideo monitors (one for each eye), VR headsets.

[0278] 3) Autostereoscopic Displays

[0279] a) Video Projector/Retroreflective screen based display systems.

[0280] b) Volumetric display systems.

[0281] c) Lenticular lens based display systems.

[0282] d) Holographic Optical Element (HOE) based display systems.

PREFERRED EMBODIMENT

[0283] In summary, the present invention provides in a preferredembodiment a system that is capable of inputting monoscopic imagesequences in a digital format, or in an analogue format in which case ananalogue to digital conversion process is involved. This image data isthen subjected to a method of image analysis whereby the monoscopicimages are compressed, if this is required for, the particularapplication.

[0284] By comparing blocks of pixels in an image, with correspondingblocks in an adjacent image, and by obtaining the minimum Mean SquareError for each block, motion within the image can be determined:

[0285] Following motion detection, regions of an image are identifiedfor similar characteristics, such as, image brightness, color, motion,pattern and edge continuity. The data is then subjected to motionanalysis in order to determine the nature of the motion in the image.This motion analysis takes the form of determining the direction, speed,type, depth and position of any motion in the image. This motion is thencategorized into a number of categories including whether the motion isa complete scene change, a simple pan, a complex pan, an object movingon a stationary background, a stationary object in front of a movingbackground, or whether there is no motion at all. Further actions arethen determined based on these categories to convert the monoscopicimages into stereoscopic image pairs suitable for viewing on anappropriate stereoscopic display device.

[0286] In the preferred embodiment, once the monoscopic images areanalyzed, if a scene change or a complex pan is detected then no furtheranalysis of that particular scene is required, rather the Field Delayand Field Delay history are both reset to zero. An object detectionprocess is then applied to the new scene in order to try and identifyobjects within that scene. Once these objects are identified, thenobject processing takes place. If no objects are identified, then theimage is passed on for further processing using forced parallax and 3Doptimization.

[0287] If the motion categorized during the image analysis is not ascene change, then further analysis of that scene is required. Iffurther analysis of that scene results in the motion being categorizedas a simple pan, then it is necessary to apply a Field Delay inaccordance with the principles of motion parallax. It is then passed onfor further processing. If the motion is not categorized as a simplepan, but rather as an object in motion on a stationary background, thenagain we have to apply a Field Delay in accordance with the principlesof motion parallax. In this regard, once the motion parallax has beenapplied, it is necessary to consider whether the objects all have auniform direction. If the objects do move in a uniform direction, thenit is passed on for further processing at a later stage. If the objectsdo not have a uniform direction, then it is necessary to perform furtherobject processing on selected objects within that scene to correct forthe Reverse 3D effect. This can be achieved through using meshdistortion and morphing techniques.

[0288] If the motion is categorized as being a stationary object on amoving background, it is then necessary to consider whether thebackground has a large variation in depth. If it does not, then we applya Field Delay with the object having priority using the principles ofmotion parallax. However, if the background does have large variation indepth, then we apply a Field Delay with the background having priorityas opposed to the object, again using the principles of motion parallax.In this case, it is then also necessary to perform further objectprocessing on the foreground object to correct for the Reverse 3D effectprior to being passed on for further processing.

[0289] If no motion is detected, then we next consider whether an objectin the scene was known from any previous motion. If this is so, then weperform object processing on that selected object. If not, then we applyan object detection process to that particular scene in order to attemptto identify any objects in it. If an object is identified, then weperform object processing on that particular object, if not, ForcedParallax and 3D Optimization is performed.

[0290] Where object processing is required, objects are identified,tagged and tracked, and then processed by using techniques of meshdistortion and morphing, object baralleling, edge enhancement,brightness modification and object rotation.

[0291] In all cases, once the motion has been categorized and theprimary techniques to convert to stereoscopic images have been applied,then a further amount of parallax or lateral shifting called forcedparallax is applied to the image. It is noted that in the preferredembodiment, forced parallax is applied to every image, not just fordepth smoothing purposes but to provide an underlying stereoscopiceffect that all images are seen as having depth behind or in front ofthe stereoscopic display device's reference plane, generally the frontof the monitor screen. The advantages of applying forced parallax arethat the system is better able to cope with changes in the category ofthe motion detected without causing sudden changes in the viewers depthperception.

[0292] Once the forced parallax has been applied to the image, the imageis then passed for 3D Optimization. Again, this is not necessary inorder to see a stereoscopic image, however the optimization does enhancethe image's depth perception by the viewer. The 3D Optimization can takein a number of forms including the addition of reference points orborders, parallax modulation, parallax smoothing and parallax adjustmentfor altering the depth sensitivity of any particular viewer. The imagecan also be optimized by modifying luminance or chrominance valuespseudo randomly so that background motion behind foreground objects canbe observed so that the depth perception is enhanced. It is alsopossible to analyze for Reverse 3D so that a viewers eyestrain isminimized. Further techniques such as shadowing, foreground andbackground fogging or blurring and edge enhancement of the image canalso be carried out in this stage.

[0293] Once the image has been optimized it is then transmitted to theappropriate display device. This transmission can take a number of formsincluding cable, co-axial, satellite or any other form of transmittingthe signal from one point to another. It is also possible that the imagecould be stored prior to being sent to a display device. The displaydevice can take on a number of forms, and only need be appropriate forthe application in hand, for example, it is possible to use existingvideo monitors with a set top device in order to separate the left andright images, increase the scan rate and to synchronize viewing glasses.Alternatively, dedicated stereoscopic displays can be used whichincorporate the use of glasses or head gear to provide the stereoscopicimages or alternatively, an auto-stereoscopic display device can beused. It is envisaged that the present invention will have applicationin theatres, cinemas, video arcades, cable or network TV, in theeducation area, particularly in the multimedia industry and in manyother areas such as theme parks and other entertainment applications.

1. A method for converting monoscopic images for viewing in threedimensions comprising: receiving monoscopic images; analyzing saidmonoscopic images to determine characteristics of the images; processingsaid monoscopic images based on the determined image characteristics;and outputting the processed images to suitable storage and/orstereoscopic display systems, wherein said analyzing further includesdetermining the motion of said monoscopic images by: dividing each imageinto a plurality of blocks, wherein corresponding blocks on an adjacentimage are offset horizontally and/or vertically; and comparing each saidblock with said corresponding blocks to find a minimum mean square errorand thereby the motion of the block.